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1.  SUMMARY 

Assessment  of  Camouflage,  Concealment  and  Deception 
(CCD)  methodologies  is  a non  trivial  problem;  conventionally 
the  only  method  has  been  to  carry  out  field  trials,  which  are 
both  expensive  and  subject  to  the  vagaries  of  the  weather.  In 
recent  years  computing  power  has  increased,  such  that  there 
are  now  many  research  programmes  using  synthetic 
environments  for  CCD  assessments.  Such  an  approach  is 
attractive;  the  user  has  complete  control  over  the  environment 
parameters  and  many  more  scenarios  can  be  investigated. 

The  UK  Defence  Evaluation  and  Research  Agency  is  currently 
developing  a synthetic  scene  generation  tool  for  assessing  the 
effectiveness  of  air  vehicle  camouflage  schemes.  The 
software  is  sufficiently  flexible  to  allow  it  to  be  used  in  a 
broader  range  of  applications,  including  full  CCD  assessment. 
The  synthetic  scene  simulation  system  (CAMEO-SIM)  has 
been  developed,  as  an  extensible  system,  to  provide  imagery 
within  the  0.4  - 14  micron  spectral  band  with  as  high  a 
physical  fidelity  as  possible.  It  consists  of  a scene  design  tool, 
an  image  generator,  which  incorporates  both  radiosity  and  ray- 
tracing processes,  and  an  experimental  trials  tool.  The  scene 
design  tool  allows  the  user  to  develop  a three-dimensional 
representation  of  the  scenario  of  interest  from  a fixed  view- 
point. Target(s)  of  interest  can  be  placed  anywhere  within  this 
3-D  representation  and  may  be  either  static  or  moving. 
Different  illumination  conditions  and  effects  of  the 
atmosphere  can  be  modelled  together  with  directional 
reflectance  effects.  The  user  has  complete  control  over  the 
level  of  fidelity  of  the  final  image.  The  output  from  the 
rendering  tool  is  a sequence  of  radiance  maps  which  may  be 
used  by  sensor  models,  or  for  experimental  trials  in  which 
observers  carry  out  target  acquisition  tasks.  The  software  also 
maintains  an  audit  trail  of  all  data  used  to  generate  a particular 
image,  both  in  terms  of  material  properties  used  and  the 
rendering  options  chosen. 

Keywords:  Scene  Simulation,  CCD  Assessment, 
Camouflage,  Concealment  & Deception 

2.  INTRODUCTION 

All  camouflage  is  a compromise.  It  is  required  to  match 
different  backgrounds,  in  different  wavebands  and  at  different 
times  of  the  year.  The  compromises  made  in  the  past  were 
determined  by  subjective  assessment  of  the  visibility  of  a 
military  asset  when  viewed  against  some  relevant  background. 
Typically,  this  assessment  was  carried  out  in  the  visible  band 
only.  Two  technologies  are  driving  the  need  for  quantitative 
CCD  assessment.  Firstly,  sensors  now  operate  throughout  a 
large  part  of  the  electromagnetic  spectrum  and  it  is  likely  that 
future  sensors  will  place  even  greater  demands  on  camouflage 
design  by  requiring  exact  spectral  matches.  Secondly,  new 
techniques  and  materials  offer  the  potential  of  increased 


effectiveness  of  camouflage  against  sensor  threats.  Cost- 
effective  and  quantitatively  correct  assessment  of  these 
techniques  and  materials  is  essential  for  future  system 
survivability. 

Synthetic  scene  generation  offers  a viable  alternative  to  field 
trials  for  the  quantitative  evaluation  of  camouflage.  We  are 
developing  a physics  based,  broadband,  scene  simulation 
toolset  called  CAMEO-SIM  that  enables  the  quantitative 
evaluation  of  both  current  and  future  camouflage.  The  same 
toolset  may  also  be  used  to  assess  concealment  and  deception 
methodologies. 

3.  STRUCTURE  OF  THE  PAPER 

Section  4,  provides  an  overview  of  the  components  that  make 
up  the  CAMEO-SIM  toolset.  Section  5 reviews  the 
verification  tests  that  have  been  carried  out  to  date,  section  6 
describes  the  validation  programme  and  section  7 presents 
conclusions. 

4.  OVERVIEW  OF  CAMEO-SIM 

The  goal  of  the  CAMEO-SIM  system  is  to  produce  synthetic, 
high  resolution,  physically  accurate  radiance  images  of  target 
vehicles  in  operational  scenarios,  at  any  wavelength  between 

0. 4. and  1 4 microns.  The  main  components  of  the  system  are 
shown  schematically  in  Figure  1.  These  are  described  in  detail 
elsewhere  [1].  The  software  was  developed  with  a scaleable 
rendering  kernel  in  which  imagery  can  be  produced  at 
different  fidelities  and  frame  rates  depending  on  the  image 
application  and  wavelength  of  operation. 

The  lowest  fidelity  mode  is  real-time  and  can  be  used  to 
develop  and  preview  the  scene  before  it  is  passed  to  a 
scaleable  two-pass  rendering  kernel  that  can  produce  high 
quality  image  streams  using  BRDF  capable  radiosity  and  ray- 
tracing algorithms.  The  first  pass  computation  models  the 
radiative  transfer  between  extended  surfaces;  ie.,  it  models 
‘soft’  shadow  effects.  The  second  pass  is  a ray  tracer  to  model 
the  effect  of  point  sources  - it  models  the  ‘hard’  shadows.  The 
images  produced  from  these  algorithms  are  then  used  in  the 
camouflage  assessment  process.  The  user  has  detailed  control 
of  the  parameters  used  in  the  calculations. 

The  rendering  equations  solved  during  real-time  operations  are 
termed  ‘local’  rendering  equations.  The  term  ‘local’  is  used  to 
acknowledge  the  fact  that  the  radiative  interactions  in  the 
scene  are  predefined.  In  order  to  run  in  real-time, 
approximations  are  made  during  the  rendering  process.  These 
include: 

1 . the  directionality  and  bi-directionality  of  the  optical 
properties  of  materials  in  the  scene  are  approximated 
using  scalar  diffuse  optical  properties,  perhaps  with  a 
simple  specular  parameter  added  to  the  reflectivity. 

2.  global  illumination  effects  are  not  accounted  for. 
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3.  geometric  occlusion  of  point  and  extended  sources  of 
radiation  is  ignored. 

4.  spectral  integration  of  the  optical  properties  with  the 
atmospherics  is  reduced  to  a multiplication  of  the  in-band 
optical  property  with  the  in-band  atmospheric  term  - 
such  an  approximation  is  only  valid  for  spectrally  grey 
materials. 

5.  the  parameterisation  of  the  atmospheric  terms  is 
simplified  - for  example  3D  variations  in  the  path 
radiance  and  transmittance  arc  ignored. 

6.  polygon  budgets  arc  restricted  - which  means  that  the 
structure  of  complex  objects  such  as  vehicles  or  trees  has 
to  be  simplified. 

To  assess  the  effectiveness  of  camouflage  schemes,  especially 
on  helicopters  flying  at  low  level  using  trees  as  natural  cover, 
these  real-time  approximations  can  impact  critically  on  the 
final  computed  air  vehicle  contrast.  Therefore  higher  fidelity, 
non  real-time  computations  must  be  used.  The  CAMFO-SIM 
rendcrers  do  not  make  the  approximations  listed  above.  The 
output  from  the  high  fidelity  rendering  kernel  is  a sequence  of 
floating  point,  2-D  radiance  maps  for  a given  waveband  or 
bands.  These  may  then  be  played  back  at  real-time  rates  for 
different  applications. 


-o- 

Image  Display  ^ . alilrati 

/ o 

T • , , , Trial  Design 

Inals  niuuuie 

o 

t ••:!••!! fin-  - 


Figure  1 Block  diagram  of  CAMEO-SIM  components 
illustrating  the  data  flow  through  the  different  processes. 


5.  ANALYTICAL  VERIFICATION  TESTS 

CAMEO-SIM  Version  1.0  is  complete  and  is  now  undergoing 
verification  and  validation.  A range  of  verification  tests  has 
been  developed  that  exercise  different  elements  of  the  high 
fidelity  rendering  equations  implemented  within  CAMEO- 
SIM.  All  the  tests  have  analytic  solutions.  Table  1 


summarises  the  test  results.  A detailed  description  of  each  lest 
is  given  in  the  following  sections. 


Test 

PA  pec  ted  result 

Calculated 

Blackbody 

radiance 

Blackbody  Radiance  - 42.89 
(8-12.5  micron  band) 

42.89 

Contrast  in 
isothermal 
environment 

Centre  pixel  radiance  - 35.23 
(8-12.5  micron  band) 

35.23 

Shadowing 

and 

a.  Radiance  of  irradiated  area  = 
5.1768 

a.  5.1768 

Blocking 

b.  Radiance  of  blocked  area  = 0.0 

c.  Radiance  of  shadowed  area  = 0.0 
(3-5  micron  hand) 

b.  0.0 

c.  0.0 

Spectral 

calculations 

Centre  pixel  radiance  (3-5  micron 
band)  = 1.49 

1.49 

Radiometric 

Radiance  variation: 

Centre:  0.31806 

calculation 

Centre  : 0.3183! 

edge:  0.0094239 

of  lighting 
effects 

edge  : 0.009424 S 

Directional 

emission 

Slope  of  radiance  along  centreline  = 
60.01  W m2  pixel’’ 

59.932  W m ! pixel-' 

Multiple 

material 

Blackbody  radiance  - 8.975 

Blackbody  radiance 
= 8.975 

Assignment 

Cirev  body  radiance  =4.4875 

Grey  body  radiance 

on  a texture 

(3-5  micron  band) 

= 4.4875 

Bi- 
directional 
reflectivity  | 

Illuminated  pixel  radiance  = 2.3 

2.3 

Small  target 
rendering 

Integrated  facet  radiant  intensity  = 
1.806  W sr'1  (3-5  micron  band) 

1 .813  W sr-1 

Table  1 Summary  of  validation  test  results.  All  values  arc 
VV  m'2  sr'1  unless  otherwise  stated. 


5.1.  Blackbody  radiance  tests 

The  purpose  of  this  test  was  to  ensure  that  the  blackbody 
radiance  is  calculated  correctly.  A one  metre  square 
uniformly  textured  facet  was  created  and  the  temperature  of 
the  facet  set  to  a known  value.  The  line  of  sight  of  the 
observer  was  centered  and  perpendicular  to  the  facet.  The 
radiance  of  a perfect  blackbody  was  calculated  and  compared 
with  the  value  computed  within  CAMEO-SIM. 

5.2.  Contrast  in  an  isothermal  environment 

The  purpose  of  this  test  is  to  ensure  that  the  correct  radiance 
contrast  is  predicted  for  isothermal  vacuum,  radiometric 
environments. 

The  skyshine  radiance  terms  are  set  to  constant  values.  A one 
metre  square  surface  is  defined  to  be  a perfect  diffuse  reflector 
and  the  line  of  sight  of  the  observer  is  centered  and 
perpendicular  to  the  facet.  The  radiance  of  the  square  is 
calculated  and  compared  with  the  value  computed  within 
CAMEO-SIM. 

5.3.  Calculation  of  shadowing  and  blocking 

Blocking  is  the  rendering  process  that  ensures  parts  of  the 
object  that  arc  not  visible  to  the  observer  due  to  obstruction  by 
another  part  are  correctly  accounted  for.  Shadowing  is  the 
rendering  process  that  ensures  parts  of  the  object  do  not  reflect 
the  point  sources  if  they  are  obscured  from  it  by  other  parts. 
This  test  has  been  designed  to  ensure  that  the  blocking  and 
shadowing  algorithms  are  working  accurately.  The  geometry 
for  this  test  is  shown  in  Figure  2 which  shows  two  square 
plates  with  the  lower  plate  100%  diffuse  reflecting  and  the  top 
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plate  black  and  at  zero  Kelvin.  The  observer  and  sun  are  at  45 
degrees  to  the  geometry.  The  radiance  of  the  illuminated 
pixels  in  the  image  is: 

N = Q p/n  (1) 

where: 

N is  the  radiance  in  W nf2  sr"1 

Q is  the  normal  incident  irradiance  W m'2=  solar 
irradiance  x cosine  of  incidence  angle 

p is  the  diffuse  reflectance  of  the  lower  plate 

The  solar  irradiance  is  set  to  a fixed  value.  The  radiance  of 
the  shadowed,  blocked  and  irradiated  areas  is  calculated  and 
compared  with  the  values  computed  within  CAMEO-SIM. 


Figure  2 Diagram  showing  the  geometry  used  to  verify 
the  blocking  and  shadowing  computations. 


5.4.  Spectral  Calculations 

The  purpose  of  this  test  specification  was  to  verify  that  the 
spectral  integrations  were  being  calculated  accurately.  To  test 
this  a defined  solar  spectral  irradiance  was  used  to  illuminate 
an  artificial  spectral  material  being  observed  with  a spectrally 
selective  sensor. 

The  spectral  variation  in  the  material  properties,  the  light 
intensity  and  the  sensor  response  is  defined.  For  the  general 
case,  the  in-band  reflected  radiance  between  the  upper  and 
lower  wavelengths  is  given  by: 

naa  = f^cos  dre{X).p\X).dx  (2) 

i s 

where: 

N5^  = in  band  radiance 

0j  = incidence  angle  between  source  and  reflector 
J(I)=  source  intensity  (W  sr'1) 

0(X)  = sensor  spectral  response 
p“(X)  = spectral  bi-directional  reflectivity 
s = distance  to  the  source 

5.5.  Radiometric  calculation  of  lighting  effects 

The  purpose  of  this  test  was  to  verify  that  the  radiometric 
effects  of  light  sources  are  being  accurately  represented.  The 
geometry  of  the  test  is  shown  in  Figure  3a  and  a plot  of 


computed  (red  line)  and  rendered  radiance  is  shown  in  Figure 
3b,  together  with  the  difference  between  the  computed  and 
rendered  radiance.  It  must  be  noted  the  analytical  solution 
assumes  radiant  intensity  is  at  the  pixel’s  centre,  but  the 
image’s  radiant  intensity  is  super  sampled  across  a pixel.  This 
will  introduce  a small  difference  to  the  analytical  solution. 

5.6.  Directional  emission  of  uniformly  textured  and  heated 
spheres 

This  test  verifies  that  the  second  pass  Tenderer  is  accounting 
for  the  directional  emissivity  correctly  when  the  object  is 
nominated  as  having  directional  optical  properties. 

Two  uniformly  textured  spheres  of  2m  diameter  are  set  to  a 
known  temperature.  For  one  of  the  spheres,  the  vertex 
normals  are  equal  to  the  facet  normal  and  for  the  other,  an 
appropriate  angle  is  chosen  for  generating  the  vertex  normals. 
Therefore,  in  the  test  both  flat  faceting  and  vertex  normal 
interpolation  in  the  second-pass  Tenderer  are  tested.  The 
variation  in  pixel  radiance  from  the  centre  of  the  sphere  to  the 
outside  edge  should  vary  linearly  (for  the  vertex  normal 
interpolated  sphere,  and  approximated  with  a stepped  variation 
for  the  flat  facet  sphere. 


A Directional 


Pixel  No 


Figure  3 a)  Lighting  effects  geometry,  b)  graph  of 
computed  and  predicted  radiance  as  a function  of  pixel 
position. 

5.7.  Textured  heated  billboard  for  testing  multiple 
material  assignments  on  a texture. 

The  purpose  of  this  test  is  to  ensure  that  textures  that  have 
been  classified  using  multiple  material  associations  and 
transparency  are  interpreted  properly  by  CAMEO-SIM.  To 
test  this  aspect  a heated  non-uniformly  textured  billboard  with 
a transparent  section  is  rendered.  A 256  x 256  texture  image 
containing  two  rectangles  and  a transparent  region  is  created. 
One  rectangle  is  classified  as  a blackbody  perfect  diffuser  and 
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set  to  a known  temperature.  The  other  rectangle  is  set  to  he  a 
greybody  perfect  diffuser  at  the  same  temperature.  A typical 
image  expected  from  this  test  is  shown  in  Figure  4. 

5.8.  Bi-directional  reflectivity  of  uniformly  textured  and 
heated  spheres. 

The  purpose  of  this  test  was  to  verify  that  CAMFO-SIM  is 
interpreting  the  bi-directional  reflectance  function  correctly. 

To  keep  the  solution  to  the  BRDF  problem  analytically 
tractable  a BRDF  file  was  used  that  represents  a grey  serni- 
spccular  retro-reflecting  BRDF  such  that 

BRDF  = — - — 0 < 30deg 

cos(i9) 

BRDF  = 0.0  6 > 30  deg 


Blackbody 


Transparent 

Region 


Figure  4 Schematic  of  the  image  produced  by  the 
textured  heated  billboard  test. 

where  0 is  the  angle  of  incidence.  Two  spheres  are  created: 
for  one  sphere  the  vertex  normals  arc  equal  lo  the  facet  normal 
and  for  the  other  sphere  an  appropriate  angle  is  chosen  for 
generating  the  vertex  normals.  The  line  of  sight  of  the 
observer  is  set  to  view  the  spheres  from  above  with  the  sun 
position  above  the  observer.  A typical  image  in  which  the 
illuminated  pixels  have  a nearly  constant  radiance  across  their 
diameters  is  shown  in  Figure  5. 


5.9.  Small  target  rendering 

The  purpose  of  this  test  was  to  ensure  that  CAMFO-SIM  is 
treating  small  targets  to  an  acceptable  accuracy;  an  essential 
requirement  for  simulating  potentially  sub-pixel  targets.  To 
test  this  requirement  an  identical  sphere  to  that  used  in  the 
BRDF  test  is  rendered  against  a simple  uniform  background. 
The  geometry  of  the  test  case  is  shown  in  Figure  6a,  and  the 
image  formed  for  this  test  case  should  be  similar  to  that  shown 
in  Figure  6b.  The  predicted  integrated  facet  radiant  intensity 

is  1.806  W m 2 sr“' . The  CAMFO-SIM  integrated  facet  radiant 
intensity  is  1.813  VV  m'2sr''  - a percentage  difference  of 
0.39%.' 
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Figure  6 a)  Geometry  used  to  carry  out  the  small  target 
test,  b)  A typical  image  produced  by  the  test. 


Figure  5 Image  resulting  from  the  bi-directional 
reflectivity  test. 


5.10.  Fit  for  purpose  imagery 

The  verification  tests  have  shown  that  CAMFO-SIM  is 
computing  the  correct  values.  I lowcvcr  rendering  times  arc 
excessively  long  if  the  complete  solution  is  calculated  - 
especially  for  a complex  structured  scene.  CAMFO-SIM  was 
designed  therefore  to  render  images  at  different  levels  of 
fidelity,  for  different  applications  - such  that  the  imagery  is  ‘fit 
for  purpose'. 

Some  example  images  for  a clear  day.  visible  band,  generated 
at  different  levels  of  fidelity  are  shown  in  Figure  7.  These 
images  were  created  using  three-dimensional  models  of  trees 
constructed  from  triangular  facets.  No  sensor  effects  have 
been  added  to  the  images. 

Similar  images  can  be  created  for  different  wavebands 
including  visible.  3 - 5 micron  and  8-12  micron  band  (Figure 
8).  A subjective  analysis  of  the  above  images  shows  that  the 
significance  of  different  effects  varies  with  waveband  and 
with  the  weather  condition  - as  expected.  For  example,  on  a 
cloudy  day  the  hard  shadows  from  the  sun  arc  not  relevant. 
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Shadows  appear  to  have  a more  significant  effect  in  images  of 
wavebands  < 5 microns  than  for  8-12  micron  band.  In  the 
lower  wavebands  the  ‘soft’  shadow  effects  produce  the  three- 
dimensional  effects  of  shadows  on  trees  which  appears  to  have 
a large  effect  on  the  contrast  structure  within  the  image,  but 
this  imposes  a heavy  computational  load  on  the  Tenderer. 

Directional  reflectance  effects  give  rise  to  glints  and  cause 
more  structure  to  be  visible  within  the  object  - this  is  going  to 
have  a very  significant  effect  on  the  spatial  contrast  structure 
within  the  image. 

The  image  in  Figure  9 shows  the  result  of  differencing  and 
histogram  equalising  a ‘low’  fidelity  and  a ‘high’  fidelity 
image.  Clearly  there  are  large  differences  that  could 
contribute  significantly  to  errors  in  target  conspicuity.  For  an 
image  to  be  ‘fit  for  purpose’  the  errors  shown  in  the  difference 
image  must  be  insignificant  for  that  particular  application. 

When  viewed  through  a sensor  the  image  resolution  will  be 
degraded  and  hence  a lower  level  of  fidelity  may  be 
acceptable.  In  addition  the  ‘real  world’  is  inherently  variable, 
so  the  images  only  have  to  be  accurate  within  the  limits  of  the 
natural  variations  whilst  still  capturing  the  spatial  and  spectral 
structure. 


Figure  7 Example  images  rendered  to  different  levels  of 
fidelity:  a)  all  surfaces  diffuse,  b)  hard  shadowing  added, 
c)  hard  and  soft  shadowing  plus  BRDF  characteristics 
applied  to  the  aircraft. 

6.  VALIDATION 
6.1.  Issues 

The  issues  surrounding  the  validation  of  any  piece  of 
simulation  software  are  often  complex  and  CAMEO-SIM  is 
no  exception.  Furthermore,  the  fact  that  CAMEO-SIM  aims 
to  physically  represent  the  real  world,  in  many 
electromagnetic  wavebands  adds  considerably  to  the 


difficulties,  since  we  still  have  neither  the  basic  databases  nor 
the  necessary  understanding  of  what  constitutes  the  real  world. 
In  addition,  since  the  whole  purpose  of  CAMEO-SIM  is  to 
represent  scenarios  that  may  not  exist  or  are  impossible  to 
document,  there  may  in  fact  be  no  equivalent  real  world.  This 
can  be  illustrated  at  the  simplest  level  by  considering  the 
geometry  and  culture  that  are  used  to  describe  a scenario.  It  is 
possible  to  achieve  an  exact  match  with  the  terrain  geometry 
by  using  detailed  map  information,  but  it  is  impossible  to 
achieve  exactly  the  same  geometry  for  the  culture  present  in 
that  terrain  (eg  tree  structure).  This  means  that  validation 
methods  that  assume  there  is  some  real  world  database  of 
measurements  that  can  be  directly  compared  with  the  output 
from  the  simulation  cannot  work.  The  validation  processes 
that  we  are  using  are,  as  a result,  somewhat  more  abstract  and 
involve  three  separate  approaches.  Firstly,  to  use  highly 
simplified  scenarios  that  can  be  both  synthesised  within 
CAMEO-SIM  and  measured. 


Figure  8 Examples  of  images  generated  in  different 
wavebands:  a)  visible  band,  b)  3 - 5 p,  c)  8 - 12  p 


Figure  9 Histogram  equalised  difference  between  a low 
fidelity  and  a high  fidelity  image  with  identical  geometry. 

These  move  one  step  away  from  the  basic  analytical 
verification  tests  described  in  section  5.  Secondly,  we  are 
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examining  whether  the  statistics  of  the  imagery  produced  by 
CAMEO-SIM  arc  consistent  with  the  statistics  of  real  world 
images.  Finally,  since  the  main  use  of  this  tool  is  for 
camouflage  assessment  we  will  use  it  to  reproduce  a real- 
world  trial.  This  will  assess  whether  the  observer 
performance,  using  the  synthetic  imagery  CAMEO-SIM 
generates,  corresponds  to  the  performance  actually  measured 
in  the  field.  F,ach  of  these  validation  approaches  is  described 
in  more  detail  below. 

6.2.  Simple  Imagery  Validation 

None  of  the  scenarios  used  in  the  tests  described  section  4 
were  real.  The  next  step  therefore  is  to  exercise  CAMF.O-SIM 
with  imagery  that  is  more  realistic.  We  plan  to  conduct  a 
series  of  trials  involving  imaging  a simple  object  viewed 
against  a uniform  background,  flic  object  is  a metal  step-like 
structure  and  is  shown  in  Figure  10.  The  object  is  sufficiently 
complex  to  enable  us  to  exercise  both  the  radiosity  and  ray- 
tracing  processes  within  the  software.  Radiation  will  be 
reflected  from  the  riser  of  the  step  down  onto  the  lower  step, 
the  object  will  generate  shadowing,  different  areas  will  heat  up 
differentially,  etc.  It  will  be  placed  in  the  open  and  viewed  by 
different  sensors  looking  down  into  the  “step"  area.  Imagery- 
will  be  gathered  in  the  visible  band,  MWIR.  and  I.WIR  under 
different  conditions.  The  same  scenario  will  be  constructed 
within  CAMEO-SIM  using  the  measured  geometry  and 
surface  material  properties  as  well  as  mctercological  data. 

Comparisons  will  be  made  between  the  in-band  radiance 
values  measured  on  the  real  object  under  different  conditions 
and  the  equivalent  values  calculated  within  CAMEO-SIM 
using  different  rendering  fidelities. 


Figure  10  Photograph  of  the  step  object  to  be  used  in 
validation  trials.  The  photograph  also  shows  how  the 
object  can  be  instrumented  with  thermocouples  for 
temperature  measurement.  (Lighter  patches  in  the  image 
are  shadows.) 

6.3.  Image  Statistics  Validation 

Many  image  metrics  have  been  proposed  |2. 3, 4,5.6],  Ideally, 
pixel  comparison  would  be  the  ultimate  method  of  comparing 
real  and  synthetic  images.  However  this  would,  excessively 
overspecify  the  accuracy  required,  not  only  because  of  the 
natural  variations  in  the  real-world,  but  also  because  it  is 


often  the  case  that  the  synthesized  imagery  does  not 
correspond  directly  to  any  part  of  the  real  world. 

Bivariate  metrics  which  compare  similar  images,  and  which 
arc  used  extensively  in  the  field  of  image  compression  (7) 
cannot  be  used  because  it  will  be  impossible  to  exactly 
recreate  the  geometric  structure  of  the  real  world  in  the 
synthetic  image  - and  not  necessary.  Application  of  such 
metrics  would  indicate  that  there  were  large  differences 
between  the  real  and  synthetic  images,  but  these  differences 
are  not  meaningful  for  this  application. 

Univariate  metrics  based  on  statistics  of  a single  image  are 
more  appropriate  as  an  ‘image  quality’  metric.  However  first 
order  statistics,  such  as  mean  and  standard  deviation  are  not  an 
appropriate  measure  for  the  spatial  structure  in  an  image 
because  two  totally  different  images  can  have  the  same  first 
order  statistics. 

Similarly  second  order  statistics,  such  as  the  power  spectrum, 
do  not  completely  describe  natural  images.  However,  it  is 
clear  that  if  our  simulation  tool  is  not  even  capturing  the  first 
order  statistics  of  the  real  world  it  has  questionable  validity. 

To  illustrate  this  we  provide  here  a comparison  we  have 
carried  out  betw  een  the  colour  characteristics  of  real  world 
images  and  equivalent  synthetic  ones  (using  a visible-band 
precursor  to  CAMF.O-SIM  called  CAM-SIM).  A set  ol  real 
world  images  obtained  by  a photographic  colorimetric  method 
(8)  provided  the  necessary  data  for  the  real  world.  These  were 
obtained  during  a trial  in  1982  in  Southern  England.  The 
scenario  consisted  of  open  grass  fields  with  scrub  and 
woodland  clumps.  There  were  no  man-made  buildings  present 
in  the  images  although  a portion  of  the  image  was  occupied  by 
a military  vehicle.  Images  were  captured  on  different  summer 
days  and  at  different  ranges  from  0.5km  to  3km.  A similar 
scenario  was  constructed  using  CAM-SIM.  Colour  statistics 
were  collected  for  the  two  types  of  image  and  the  resulting 
histograms  are  show  n in  Figure  ll.lt  is  clear  that  the  mean 
colour,  expressed  as  chromaticitv.  of  the  two  sets  of  imagery 
is  similar,  but  that  the  distributions  and  ranges  of  the  synthetic 
and  real  imagery  are  quite  different.  For  certain  applications 
these  differences  may  be  critical.  Similar  comparisons  on  a 
range  of  first  and  second  order  statistics  are  planned  for 
CAMEO-SIM. 

Higher  order  statistics  which  can  capture  phase  information 
arc  likely  to  be  the  most  appropriate  metrics  to  investigate. 
However  these  are  complex  and  hence  difficult  to  interpret 
and  apply.  Certain  types  of  Neural  networks  (Independent 
Component  Analysis  networks)  naturally  capture  some  of  the 
features  of  higher  order  statistics  after  training.  These 
networks  can  be  trained  on  real  imagery  and  then  used  on 
synthetic  images  to  find  if  the  same  characteristics  are 
detected.  Therefore  it  is  believed  that  neural  networks  would 
be  an  appropriate  statistical  solution  for  this  project. 

This  work  w ill  continue  and  different  metrics  used  to  analyse 
the  synthetic  image  fidelity  so  that  appropriate  images  arc 
used  for  different  applications. 

6.4.  Performance  Validation 

The  third  element  of  the  validation  programme  will  involve 
comparisons  of  performance.  The  aim  is  to  reproduce  a real 
trial  using  imagery  generated  by  CAMEO-SIM. 

A flight  trial  was  conducted  in  1974  to  compare  a light  grey 
camouflage  scheme  with  a dark  grey  /green  scheme  on  UK 
Royal  Air  Force  strike  aircraft.  The  comparison  was  made  by 
flying  aircraft  painted  in  the  two  schemes  against  ground 
based  observers  under  a variety  of  meteorological  conditions 
and  recording  the  detection  ranges  obtained.  Observation  w'as 
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made  with  the  unaided  eye  and  with  magnifying  sights  (X5 
and  X10).  The  nature  of  the  trial  required  the  aircraft  to  fly 
accurate  straight-line  tracks  of  about  20kms  on  a variety  of 
headings  and  at  low  level. 


Figure  11  CIE  (x,y)  chromaticity  histograms,  a) 
distribution  for  real  imagery,  b)  distribution  for  synthetic 
imagery. 

The  following  conditions  and  variables  were  selected: 

1 . Observers  used  unaided  viewing,  x5  and  x 1 0 optical 
sights. 

2.  Limited  search  - the  observers  knew  the  approximate 
height  and  direction  of  approach  but  were  not  accurately 
laid-on.  Search  using  the  magnifying  sights  was  within 
the  field-of-view  along  a fixed  sight-line 

3.  The  trial  was  conducted  under  three  types  of  sky 
condition  - clear  sky,  2/8-4/8  broken  cloud  and  uniform 
overcast.  Visibility  was  at  least  10km. 

4.  Three  relative  aircraft/observer/sun  positions  were 
studies:  sun  directly  behind  the  observer  as  he  viewed  the 
approaching  aircraft,  sun  at  60°  to  the  approach  path 
shining  onto  the  front  of  the  aircraft  and  sun  at  45°  to  the 
approach  path  shining  onto  the  rear  of  the  aircraft. 

5.  Approaches  were  made  head-on  to  the  observers  and  to 
cross  1000m  to  left  and  right  of  the  observers. 
Approaches  began  20km  from  the  observers. 

6.  Aircraft  were  200m  above  ground  level,  speed  300  knots. 

There  are  27  possible  different  combinations  of  the  above 
variables,  but  because  sun  position  is  not  relevant  under  a 
uniformly  overcast  sky  and  crossing  to  left  and  right  was  not 
required  when  the  sun  was  directly  behind  observers  the  actual 
number  of  different  conditions  reduces  to  18.  Each  aircraft 
made  6 approach  runs  per  condition  tested  - 1 8 sorties  were 
required  to  complete  the  trial. 


Both  experienced  and  inexperienced  observers  were  used. 
Experienced  observers  had  a visual  acuity  of  6/6  or  better  with 
normal  colour  and  binocular  vision.  Inexperienced  observers 
were  tested  using  a car  number  plate  reading  test  at  a range  of 
30m. 

Cloud  conditions  generally  and  in  the  direction  of  view  were 
noted.  Colour  of  background  sky  was  estimated  by 
comparison  with  a set  of  Munsell  colour  analysis  charts. 
Background  sky  luminance  was  measured  with  a Weston 
Master  IV  light  meter. 

Detection  ranges  were  obtained  by  timing  each  approach.  The 
observers,  with  stop  watches  were  given  a signal  as  the  aircraft 
crossed  the  IP  and  started  their  watches,  they  then  stopped 
their  watches  individually  upon  detecting  the  aircraft.  During 
every  approach  the  aircraft  was  also  timed  between  two  timing 
marks  enabling  its  ground  speed  to  be  accurately  determined. 
This  ground  speed,  the  distance  between  the  observers  and  the 
IP,  and  the  time  to  detection  were  then  used  to  calculate 
detection  range. 

The  validation  exercise  will  consist  of  a laboratory  simulation 
of  this  trial.  Clearly  attempting  to  reproduce  exactly  the  same 
conditions  is  impossible  therefore  a much  simplified  trial 
design  will  be  used.  Visible  band  image  sequences  of  the 
scenario  containing  aircraft  in  the  two  different  camouflage 
schemes  will  be  generated  using  CAMEO-SIM.  These  will  be 
played  back  to  observers  viewing  a calibrated  colour  monitor 
from  a fixed  distance  (images  are  rendered  to  be  viewed  from 
a predetermined  distance)  as  shown  in  Figure  12.  Anyone 
sequence  will  consist  of  an  image  of  an  aircraft  “painted”  in 
one  of  two  colours,  which  will  fly  along  a randomised  track 
towards  the  observer.  The  observer  task  will  be  to  press  a 
mouse  button  when  the  aircraft  is  detected  and  then  to 
designate  the  aircraft  location  on  the  screen  using  a cursor. 


Figure  12  Experimental  arrangement.  The  observer  is 
maintained  at  a fixed  distance  from  the  calibrated  display 
used  to  present  the  test  imagery.  An  additional  projector 
is  used  to  provide  a controlled  adaptation  state. 

Response  time  will  provide  the  detection  range  just  as  in  the 
real  trial.  Additionally,  this  experiment  will  provide 
information  on  accuracy  of  designation.  These  data  will  be 
compared  with  the  results  from  the  actual  trial.  Because  there 
are  so  many  differences  between  the  real  and  the  simulated 
trial  conditions  only  relative  performance  will  be  compared. 
Observers  will  be  screened  for  colour  vision  and  acuity  prior 
to  taking  part  in  the  trial.  A mean  level  of  adaptation  will  be 
maintained  by  surrounding  the  display  with  a white  screen  if 
necessary. 
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7.  CONCLUSIONS 

CAMEO-SIM  can  generate  imagery  between  0.4  and  14 
microns  to  different  levels  of  fidelity,  to  allow  a trade-off 
between  accuracy  and  rendering  time.  The  software  has 
undergone  a range  of  verification  tests  to  show  that  the  correct 
values  are  computed.  A programme  of  validation  is  now 
under  way  to  ensure  that  meaningful  results  are  obtained  using 
the  software  tools.  This  programme  will  address  three 
different  aspects  of  the  synthetic  imagery  - statistics,  real- 
world  comparisons  and  performance  prediction.  Research 
work  is  in  progress  to  quantify  the  significance  of  different 
effects  such  as  shadows,  in  different  wavebands.  The 
functionality  of  CAMEO-SIM  will  be  extended  as  part  of  an 
ongoing  research  programme  to  include  multi-processor 
rendering  capability,  and  simple  multi-spectra!  image  display, 
moving  sensors  and  thermal  shadows. 
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